End-to-End Video Question-Answer Generation With Generator-Pretester Network

We study a novel task, Video Question-Answer Generation (VQAG), for challenging Question Answering (Video QA) task in multimedia. Due to expensive data annotation costs, many widely used, large-scale QA datasets such as Video-QA, MSVD-QA and MSRVTT-QA are automatically annotated using Caption (CapQG) which inputs captions instead of the video itself. As neither fully represent video, nor they always practically available, it is crucial generate question-answer pairs based on via (VQAG). Existing video-to-text (V2T) approaches, despite taking input, only question alone. In this work, we propose model Generator-Pretester Network that focuses two components: (1) The Joint Generator (JQAG) generates with its corresponding answer allow “Answering” training. (2) Pretester (PT) verifies generated by trying checks pretested both model’s proposed ground truth answer. evaluate our system available human-annotated achieves state-of-the-art generation performances. Furthermore, can surpass some supervised baselines. pre-training strategy, outperform CapQG transfer learning approaches when employing semi-supervised (20%) or data. These experimental results suggest perspectives